
 <!DOCTYPE HTML>
<html lang="zh_CN">
<head>
  <meta charset="UTF-8">
  
    <title>Golden Pigeon&#39;s Blog</title>
    <meta name="viewport" content="width=device-width, initial-scale=1,user-scalable=no">
    
    <meta name="author" content="Golden Pigeon">
    

    
    <meta property="og:type" content="website">
<meta property="og:title" content="Golden Pigeon&#39;s Blog">
<meta property="og:url" content="http:&#x2F;&#x2F;yoursite.com&#x2F;index.html">
<meta property="og:site_name" content="Golden Pigeon&#39;s Blog">
<meta property="og:locale" content="zh_CN">
<meta property="article:author" content="Golden Pigeon">
<meta name="twitter:card" content="summary">

    
    <link rel="alternative" href="/atom.xml" title="Golden Pigeon&#39;s Blog" type="application/atom+xml">
    
    
    <link rel="icon" href="/img/favicon.ico">
    
    
    <link rel="apple-touch-icon" href="/img/ios.jpg">
    <link rel="apple-touch-icon-precomposed" href="/img/ios.jpg">
    
    
<link rel="stylesheet" href="/css/style.css">
<link rel="stylesheet" href="/%02.css">
<link rel="stylesheet" href="/.css">

<meta name="generator" content="Hexo 4.1.0"></head>

  <body>
    <header>
      
<div>
		
			<div id="imglogo">
				<a href="/"><img src="/img/logo.png" alt="Golden Pigeon&#39;s Blog" title="Golden Pigeon&#39;s Blog"/></a>
			</div>
			
			<div id="textlogo">
				<h1 class="site-name"><a href="/" title="Golden Pigeon&#39;s Blog">Golden Pigeon&#39;s Blog</a></h1>
				<h2 class="blog-motto">GP</h2>
			</div>
			<div class="navbar"><a class="navbutton navmobile" href="#" title="Menu">
			</a></div>
			<nav class="animated">
				<ul>
					<ul>
					 
						<li><a href="/">Home</a></li>
					
						<li><a href="/archives">Archives</a></li>
					
						<li><a href="/about">About</a></li>
					
					<li>
 					
					<form class="search" action="//google.com/search" method="get" accept-charset="utf-8">
						<label>Search</label>
						<input type="search" id="search" name="q" autocomplete="off" maxlength="20" placeholder="Search" />
						<input type="hidden" name="q" value="site:yoursite.com">
					</form>
					
					</li>
				</ul>
			</nav>			
</div>
    </header>
    <div id="container">
      <div id="main">

   
    
    <article class="post-expand post" itemprop="articleBody"> 
        <header class="article-info clearfix">
  <h1 itemprop="name">
    
      <a href="/2020/01/13/weekly-report-1-Jan-6th-Jan-12th/" title="weekly_report_1(Jan. 6th ~ Jan. 12th)" itemprop="url">weekly_report_1(Jan. 6th ~ Jan. 12th)</a>
  </h1>
  <p class="article-author">By
       
		<a href="/about" title="Golden Pigeon" target="_blank" itemprop="author">Golden Pigeon</a>
		
  <p class="article-time">
    <time datetime="2020-01-13T07:18:19.000Z" itemprop="datePublished"> Published 2020-01-13</time>
    
  </p>
</header>
    <div class="article-content">
        
        <h1 id="2020年寒假第一周周报-Jan-6th-Jan-12th"><a href="#2020年寒假第一周周报-Jan-6th-Jan-12th" class="headerlink" title="2020年寒假第一周周报(Jan. 6th ~ Jan. 12th)"></a>2020年寒假第一周周报(Jan. 6th ~ Jan. 12th)</h1><h2 id="本周学习内容"><a href="#本周学习内容" class="headerlink" title="本周学习内容"></a>本周学习内容</h2><h3 id="线性代数"><a href="#线性代数" class="headerlink" title="线性代数"></a>线性代数</h3><hr>
<ol>
<li>矩阵的基本运算：<br> -加减：相同维度的矩阵对应元素加减<br> -标量积：矩阵的每个元素分别与标量相乘<br> -点乘：行向量和列向量相乘得到一个标量或者一个1*1矩阵，依此规律得到结果<br> -转置：A(ij)=A^T(ji)<br> -逆矩阵：A*A^(-1)=I<br> -单位矩阵：AI=IA=AI</li>
<li>矩阵的行列式<br> -行列式=0，则矩阵为奇异矩阵，不可逆，否则为非奇异矩阵，可逆。</li>
</ol>
<h3 id="机器学习"><a href="#机器学习" class="headerlink" title="机器学习"></a>机器学习</h3><hr>
<h4 id="线性回归"><a href="#线性回归" class="headerlink" title="线性回归"></a>线性回归</h4><hr>
<h5 id="假设函数-Hypothesis"><a href="#假设函数-Hypothesis" class="headerlink" title="假设函数(Hypothesis)"></a>假设函数(Hypothesis)</h5><hr>
<p><img src="" alt="假设函数"></p>
<h5 id="代价函数-Cost-Function"><a href="#代价函数-Cost-Function" class="headerlink" title="代价函数(Cost Function)"></a>代价函数(Cost Function)</h5><hr>
<p><img src="" alt="代价函数"></p>
<h5 id="梯度下降-Gradient-Descent"><a href="#梯度下降-Gradient-Descent" class="headerlink" title="梯度下降(Gradient Descent)"></a>梯度下降(Gradient Descent)</h5><hr>
<p><img src="" alt="更新参量"></p>
<p><img src="" alt="向量化"></p>
<h4 id="逻辑回归"><a href="#逻辑回归" class="headerlink" title="逻辑回归"></a>逻辑回归</h4><hr>
<h5 id="假设函数"><a href="#假设函数" class="headerlink" title="假设函数"></a>假设函数</h5><hr>
<p><img src="" alt="假设函数"></p>
<h5 id="代价函数"><a href="#代价函数" class="headerlink" title="代价函数"></a>代价函数</h5><hr>
<p><img src="" alt="代价函数"></p>
<h5 id="梯度下降"><a href="#梯度下降" class="headerlink" title="梯度下降"></a>梯度下降</h5><hr>
<p>同线性回归</p>
<h4 id="正则化"><a href="#正则化" class="headerlink" title="正则化"></a>正则化</h4><hr>
<p>在代价函数中加入惩罚项，防止高阶特征的参数过大导致过度拟合</p>
<h2 id="本周工作总结"><a href="#本周工作总结" class="headerlink" title="本周工作总结"></a>本周工作总结</h2><p>本周我主要完成了线性回归和爬虫程序的制作，通过一遍遍的尝试，我使我的程序在变快的同时也变得更加精确。这不仅磨炼了我的毅力，也使我获得了更多的经验。下周继续努力！</p>
<h2 id="下周学习安排"><a href="#下周学习安排" class="headerlink" title="下周学习安排"></a>下周学习安排</h2><ul>
<li>继续学习线性代数</li>
<li>完成神经网络基础的学习并完成相关程序的编写</li>
<li>每天学习英语</li>
</ul>
<hr>
<p>#END</p>

        
        
        <p class="article-more-link">
          
       </p>
    </div>
    <footer class="article-footer clearfix">
<div class="article-catetags">


</div>




<div class="comments-count">
	
</div>

</footer>


    </article>






   
    
    <article class="post-expand post" itemprop="articleBody"> 
        <header class="article-info clearfix">
  <h1 itemprop="name">
    
      <a href="/2020/01/08/Obtaining-Data-From-the-MOT-Official-Website-by-Python-Module-Urllib-and-Re/" title="Obtaining Data From the MOT Official Website by Python Module Urllib and Re" itemprop="url">Obtaining Data From the MOT Official Website by Python Module Urllib and Re</a>
  </h1>
  <p class="article-author">By
       
		<a href="/about" title="Golden Pigeon" target="_blank" itemprop="author">Golden Pigeon</a>
		
  <p class="article-time">
    <time datetime="2020-01-08T03:47:18.000Z" itemprop="datePublished"> Published 2020-01-08</time>
    
  </p>
</header>
    <div class="article-content">
        
        <h2 id="基于Python-Urllib库和Re库的爬虫应用"><a href="#基于Python-Urllib库和Re库的爬虫应用" class="headerlink" title="基于Python Urllib库和Re库的爬虫应用"></a>基于Python Urllib库和Re库的爬虫应用</h2><p>获取交通部官网2019年交通固定资产投资额等数据并将其存入Excel表格中</p>
<h3 id="分析所需数据的位置及其上下文特征"><a href="#分析所需数据的位置及其上下文特征" class="headerlink" title="分析所需数据的位置及其上下文特征"></a>分析所需数据的位置及其上下文特征</h3><p>进入<a href="http://www.mot.gov.cn/shuju/" target="_blank" rel="noopener">交通部官网数据网站</a>。<br>通过右键检查表格元素，可以发现，每一个表格都被封装在一个html文件中，他们的网址位于网页源代码的872~893行，而他们的标题则位于864~867行。<br>代码如下：</p>
<figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span></pre></td><td class="code"><pre><span class="line">        <span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"fl w100 mod_spsj mod1_kshsj_sj mart15"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">2</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;<span class="name">ul</span> <span class="attr">id</span>=<span class="string">"myTabs"</span> <span class="attr">class</span>=<span class="string">"nav nav-tabs hidden-sm hidden-xs"</span> <span class="attr">role</span>=<span class="string">"tablist"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">3</span></pre></td><td class="code"><pre><span class="line">				<span class="tag">&lt;<span class="name">li</span> <span class="attr">class</span>=<span class="string">"sj_w180625"</span>&gt;</span><span class="tag">&lt;<span class="name">b</span>&gt;</span>可视化数据<span class="tag">&lt;/<span class="name">b</span>&gt;</span><span class="tag">&lt;/<span class="name">li</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">4</span></pre></td><td class="code"><pre><span class="line">				<span class="tag">&lt;<span class="name">li</span> <span class="attr">class</span>=<span class="string">"active"</span>&gt;</span><span class="tag">&lt;<span class="name">a</span> <span class="attr">href</span>=<span class="string">"#jtgdzctzwce"</span> <span class="attr">role</span>=<span class="string">"tab"</span> <span class="attr">data-toggle</span>=<span class="string">"tab"</span>&gt;</span>交通固定资产投资完成额<span class="tag">&lt;/<span class="name">a</span>&gt;</span><span class="tag">&lt;/<span class="name">li</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">5</span></pre></td><td class="code"><pre><span class="line">				<span class="tag">&lt;<span class="name">li</span>&gt;</span><span class="tag">&lt;<span class="name">a</span> <span class="attr">href</span>=<span class="string">"#yyxkhysl"</span> <span class="attr">role</span>=<span class="string">"tab"</span> <span class="attr">data-toggle</span>=<span class="string">"tab"</span>&gt;</span>营业性客货运输量<span class="tag">&lt;/<span class="name">a</span>&gt;</span><span class="tag">&lt;/<span class="name">li</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">6</span></pre></td><td class="code"><pre><span class="line">				<span class="tag">&lt;<span class="name">li</span>&gt;</span><span class="tag">&lt;<span class="name">a</span> <span class="attr">href</span>=<span class="string">"#cskyl"</span> <span class="attr">role</span>=<span class="string">"tab"</span> <span class="attr">data-toggle</span>=<span class="string">"tab"</span>&gt;</span>城市客运量<span class="tag">&lt;/<span class="name">a</span>&gt;</span><span class="tag">&lt;/<span class="name">li</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">7</span></pre></td><td class="code"><pre><span class="line">				<span class="tag">&lt;<span class="name">li</span>&gt;</span><span class="tag">&lt;<span class="name">a</span> <span class="attr">href</span>=<span class="string">"#gkscl"</span> <span class="attr">role</span>=<span class="string">"tab"</span> <span class="attr">data-toggle</span>=<span class="string">"tab"</span>&gt;</span>港口生产量<span class="tag">&lt;/<span class="name">a</span>&gt;</span><span class="tag">&lt;/<span class="name">li</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">8</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;/<span class="name">ul</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">9</span></pre></td><td class="code"><pre><span class="line">......</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">10</span></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"list-group tab-content mod1_kshsj1_sj"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">11</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;<span class="name">div</span> <span class="attr">role</span>=<span class="string">"tabpanel"</span> <span class="attr">class</span>=<span class="string">"tab-pane active"</span> <span class="attr">id</span>=<span class="string">"jtgdzctzwce"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">12</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"fl w100 mod1_kshsj1_a_sj"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">13</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;<span class="name">IFRAME</span> <span class="attr">align</span>=<span class="string">"top"</span> <span class="attr">width</span>=<span class="string">"1110px"</span> <span class="attr">height</span>=<span class="string">"820px"</span> <span class="attr">frameborder</span>=<span class="string">"0"</span> <span class="attr">src</span>=<span class="string">"http://106.3.149.173/CatsCategory/listGLJCSJ5?tableName=cats_jtysybsjk_csky&amp;pubflag=1&amp;code=C13"</span> <span class="attr">scrolling</span>=<span class="string">"no"</span> &gt;</span><span class="tag">&lt;/<span class="name">IFRAME</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">14</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">15</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">16</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;<span class="name">div</span> <span class="attr">role</span>=<span class="string">"tabpanel"</span> <span class="attr">class</span>=<span class="string">"tab-pane"</span> <span class="attr">id</span>=<span class="string">"yyxkhysl"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">17</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"fl w100 mod1_kshsj1_a_sj"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">18</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;<span class="name">IFRAME</span> <span class="attr">align</span>=<span class="string">"top"</span> <span class="attr">width</span>=<span class="string">"1110px"</span> <span class="attr">height</span>=<span class="string">"820px"</span> <span class="attr">frameborder</span>=<span class="string">"0"</span> <span class="attr">src</span>=<span class="string">"http://106.3.149.173/CatsCategory/listGLJCSJ5?tableName=cats_jtysybsjk_gksc&amp;pubflag=1&amp;code=C13"</span> <span class="attr">scrolling</span>=<span class="string">"no"</span> &gt;</span><span class="tag">&lt;/<span class="name">IFRAME</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">19</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">20</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">21</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;<span class="name">div</span> <span class="attr">role</span>=<span class="string">"tabpanel"</span> <span class="attr">class</span>=<span class="string">"tab-pane"</span> <span class="attr">id</span>=<span class="string">"cskyl"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">22</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"fl w100 mod1_kshsj1_a_sj"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">23</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;<span class="name">IFRAME</span> <span class="attr">align</span>=<span class="string">"top"</span> <span class="attr">width</span>=<span class="string">"1110px"</span> <span class="attr">height</span>=<span class="string">"820px"</span> <span class="attr">frameborder</span>=<span class="string">"0"</span> <span class="attr">src</span>=<span class="string">"http://106.3.149.173/CatsCategory/listGLJCSJ5?tableName=cats_jtysybsjk_jtgdzctzwce&amp;pubflag=1&amp;code=C13"</span> <span class="attr">scrolling</span>=<span class="string">"no"</span> &gt;</span><span class="tag">&lt;/<span class="name">IFRAME</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">24</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">25</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">26</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;<span class="name">div</span> <span class="attr">role</span>=<span class="string">"tabpanel"</span> <span class="attr">class</span>=<span class="string">"tab-pane"</span> <span class="attr">id</span>=<span class="string">"gkscl"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">27</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">"fl w100 mod1_kshsj1_a_sj"</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">28</span></pre></td><td class="code"><pre><span class="line">			<span class="tag">&lt;<span class="name">IFRAME</span> <span class="attr">align</span>=<span class="string">"top"</span> <span class="attr">width</span>=<span class="string">"1110px"</span> <span class="attr">height</span>=<span class="string">"820px"</span> <span class="attr">frameborder</span>=<span class="string">"0"</span> <span class="attr">src</span>=<span class="string">"http://106.3.149.173/CatsCategory/listGLJCSJ5?tableName=cats_jtysybsjk_yyxkhysl&amp;pubflag=1&amp;code=C13"</span> <span class="attr">scrolling</span>=<span class="string">"no"</span> &gt;</span><span class="tag">&lt;/<span class="name">IFRAME</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">29</span></pre></td><td class="code"><pre><span class="line">		<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">30</span></pre></td><td class="code"><pre><span class="line">	<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">31</span></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span></pre></td></tr></table></figure>

<p>可见，上述网址前都具有属性<code>class=&quot;fl w100 mod1_kshsj1_a_sj&quot;</code>故可以使用正则表达式<code>&lt;div class=&quot;fl w100 mod1_kshsj1_a_sj&quot;&gt;.*?src=&quot;(.*?)&quot;\s.*?&lt;/div&gt;</code>进行识别。<br>标题与此类似，不再赘述。  </p>
<p>进入每一个表格的源代码，发现每一格中的数据都被<code>&lt;td&gt;</code>和<code>&lt;/td&gt;</code>所包围，故可以使用正则表达式<code>&lt;td&gt;(.*?)&lt;/td&gt;</code>识别。表头与之类似。  </p>
<p>分析好了上下文，就可以进入编程环节了。  </p>
<h3 id="获取源代码"><a href="#获取源代码" class="headerlink" title="获取源代码"></a>获取源代码</h3><hr>
<p>使用<code>urllib</code>库中的<code>urllib.request.urlopen(url)</code>方法获取网页信息该方法接受一个字符串，返回一个<code>http.client.HTTPResponse</code>对象，该对象的read()方法返回一个字符串，即网页的源代码。通过<code>decode(&#39;utf-8&#39;)</code>方法使源代码按照UTF-8标准编码。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span></pre></td><td class="code"><pre><span class="line">response = urllib.request.urlopen(url)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">2</span></pre></td><td class="code"><pre><span class="line">    src = response.read().decode(<span class="string">'utf-8'</span>)</span></pre></td></tr></table></figure>

<h3 id="匹配字符串"><a href="#匹配字符串" class="headerlink" title="匹配字符串"></a>匹配字符串</h3><hr>
<p>通过re库中的<code>re.compile()</code>方法将正则表达式编译，并在第二个参数位置加上<code>re.S</code>以确保查找不会被换行符所中断。使用<code>re.findall()</code>方法找出源代码中所有符合匹配模式的字符串，并将其以列表的形式储存。为方便编写代码，我编写了一个<code>get_data()</code>函数整合了这个过程</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_data</span><span class="params">(url, pattern)</span>:</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">2</span></pre></td><td class="code"><pre><span class="line">    response = urllib.request.urlopen(url)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">3</span></pre></td><td class="code"><pre><span class="line">    src = response.read().decode(<span class="string">'utf-8'</span>)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">4</span></pre></td><td class="code"><pre><span class="line">    results = re.findall(pattern, src)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">5</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">return</span> results</span></pre></td></tr></table></figure>

<h3 id="储存数据"><a href="#储存数据" class="headerlink" title="储存数据"></a>储存数据</h3><hr>
<p>通过<code>openpyxl</code>库储存数据。<br>我编写了<code>fill_sheet()</code>函数以实现这一过程。该函数接收文件对象，工作表名，要填写数据的区域左上角和右下角的坐标，以及数据来源和开始位置(默认为0)，填写表格后返回最后填写的一个数据的下一位置。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fill_sheet</span><span class="params">(workbook, sheetname, pos1, pos2, src_text, reading_pos=<span class="number">0</span>)</span>:</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">2</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">if</span> sheetname <span class="keyword">not</span> <span class="keyword">in</span> workbook:  <span class="comment"># 若工作表不存在，则创建一个</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">3</span></pre></td><td class="code"><pre><span class="line">        workbook.create_sheet(sheetname)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">4</span></pre></td><td class="code"><pre><span class="line">    sh = workbook[sheetname]</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">5</span></pre></td><td class="code"><pre><span class="line">    workbook.close() <span class="comment"># 关闭文件后才能写入</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">6</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(pos1[<span class="number">0</span>], pos2[<span class="number">0</span>]+<span class="number">1</span>):</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">7</span></pre></td><td class="code"><pre><span class="line">        <span class="keyword">for</span> j <span class="keyword">in</span> range(pos1[<span class="number">1</span>], pos2[<span class="number">1</span>]+<span class="number">1</span>):</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">8</span></pre></td><td class="code"><pre><span class="line">            sh.cell(row=i, column=j, value=src_text[reading_pos])  <span class="comment"># 在相应单元格中写入数据</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">9</span></pre></td><td class="code"><pre><span class="line">            reading_pos += <span class="number">1</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">10</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">return</span> reading_pos</span></pre></td></tr></table></figure>

<h3 id="完整程序"><a href="#完整程序" class="headerlink" title="完整程序"></a>完整程序</h3><hr>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> urllib.request</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">2</span></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> re</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">3</span></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> openpyxl</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">4</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">5</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">6</span></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_data</span><span class="params">(url, pattern)</span>:</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">7</span></pre></td><td class="code"><pre><span class="line">    response = urllib.request.urlopen(url)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">8</span></pre></td><td class="code"><pre><span class="line">    src = response.read().decode(<span class="string">'utf-8'</span>)  <span class="comment"># 获取源代码</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">9</span></pre></td><td class="code"><pre><span class="line">    results = re.findall(pattern, src)  <span class="comment"># 找出所有匹配的结果</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">10</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">return</span> results</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">11</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">12</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">13</span></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fill_sheet</span><span class="params">(workbook, sheetname, pos1, pos2, src_text, reading_pos=<span class="number">0</span>)</span>:</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">14</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">if</span> sheetname <span class="keyword">not</span> <span class="keyword">in</span> workbook:  <span class="comment"># 如果工作表不存在，则创建一个</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">15</span></pre></td><td class="code"><pre><span class="line">        workbook.create_sheet(sheetname)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">16</span></pre></td><td class="code"><pre><span class="line">    sh = workbook[sheetname]  <span class="comment"># 获取工作表对象</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">17</span></pre></td><td class="code"><pre><span class="line">    workbook.close()  <span class="comment"># 关闭文件后才能写入</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">18</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(pos1[<span class="number">0</span>], pos2[<span class="number">0</span>]+<span class="number">1</span>):</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">19</span></pre></td><td class="code"><pre><span class="line">        <span class="keyword">for</span> j <span class="keyword">in</span> range(pos1[<span class="number">1</span>], pos2[<span class="number">1</span>]+<span class="number">1</span>):</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">20</span></pre></td><td class="code"><pre><span class="line">            sh.cell(row=i, column=j, value=src_text[reading_pos])  <span class="comment"># 在对应单元格中写入数据</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">21</span></pre></td><td class="code"><pre><span class="line">            reading_pos += <span class="number">1</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">22</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">return</span> reading_pos</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">23</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">24</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">25</span></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">26</span></pre></td><td class="code"><pre><span class="line">    url = <span class="string">'http://www.mot.gov.cn/shuju/'</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">27</span></pre></td><td class="code"><pre><span class="line">    <span class="comment"># 编译各种所需的正则表达式</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">28</span></pre></td><td class="code"><pre><span class="line">    pattern_get_graph_sites = re.compile(<span class="string">r'&lt;div class="fl w100 mod1_kshsj1_a_sj"&gt;.*?src="(.*?)"\s.*?&lt;/div&gt;'</span>, re.S)  <span class="comment"># 图表网址</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">29</span></pre></td><td class="code"><pre><span class="line">    pattern_get_datas = re.compile(<span class="string">'&lt;td&gt;(.*?)&lt;/td&gt;'</span>, re.S)  <span class="comment"># 表中数据</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">30</span></pre></td><td class="code"><pre><span class="line">    pattern_get_graph_headers = re.compile(<span class="string">'&lt;th style="width:120px"&gt;(.*?)&lt;/th&gt;'</span>, re.S)  <span class="comment"># 表头</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">31</span></pre></td><td class="code"><pre><span class="line">    pattern_get_graph_names_block = re.compile(<span class="string">'&lt;div class="fl w100 mod_spsj mod1_kshsj_sj mart15"&gt;(.*?)&lt;/ul&gt;'</span>, re.S)  <span class="comment"># 标题代码区块</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">32</span></pre></td><td class="code"><pre><span class="line">    pattern_get_graph_names = re.compile(<span class="string">'&lt;a.*?&gt;(.*?)&lt;/a&gt;'</span>, re.S)  <span class="comment"># 图表标题</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">33</span></pre></td><td class="code"><pre><span class="line">    <span class="comment"># 匹配数据</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">34</span></pre></td><td class="code"><pre><span class="line">    graph_names_block = get_data(url, pattern_get_graph_names_block)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">35</span></pre></td><td class="code"><pre><span class="line">    graph_sites = get_data(url, pattern_get_graph_sites)</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">36</span></pre></td><td class="code"><pre><span class="line">    graph_names = re.findall(pattern_get_graph_names, graph_names_block[<span class="number">0</span>])</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">37</span></pre></td><td class="code"><pre><span class="line">    datas = []</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">38</span></pre></td><td class="code"><pre><span class="line">    graph_headers = []</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">39</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">for</span> graph_site <span class="keyword">in</span> graph_sites:</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">40</span></pre></td><td class="code"><pre><span class="line">        datas.append(get_data(graph_site, pattern_get_datas))</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">41</span></pre></td><td class="code"><pre><span class="line">        graph_headers.append(get_data(graph_site, pattern_get_graph_headers))</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">42</span></pre></td><td class="code"><pre><span class="line">    <span class="comment"># 储存数据</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">43</span></pre></td><td class="code"><pre><span class="line">    wb = openpyxl.Workbook()</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">44</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">0</span>, len(graph_names)):</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">45</span></pre></td><td class="code"><pre><span class="line">        len_gh = len(graph_headers[i])</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">46</span></pre></td><td class="code"><pre><span class="line">        fill_sheet(wb, graph_names[i], (<span class="number">1</span>, <span class="number">1</span>), (<span class="number">1</span>, len_gh), graph_headers[i])</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">47</span></pre></td><td class="code"><pre><span class="line">        fill_sheet(wb, graph_names[i], (<span class="number">2</span>, <span class="number">1</span>), (int(len(datas[i])/len_gh+<span class="number">1</span>), len_gh), datas[i])</span></pre></td></tr><tr><td class="gutter"><pre><span class="line">48</span></pre></td><td class="code"><pre><span class="line">    <span class="keyword">del</span> wb[<span class="string">'Sheet'</span>]  <span class="comment"># 删除默认工作表</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">49</span></pre></td><td class="code"><pre><span class="line">    wb.save(<span class="string">'results.xlsx'</span>)  <span class="comment"># 保存文件</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">50</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">51</span></pre></td><td class="code"><pre><span class="line"></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">52</span></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:  <span class="comment"># 执行主函数</span></span></pre></td></tr><tr><td class="gutter"><pre><span class="line">53</span></pre></td><td class="code"><pre><span class="line">    main()</span></pre></td></tr></table></figure>

<h3 id="运行结果"><a href="#运行结果" class="headerlink" title="运行结果"></a>运行结果</h3><hr>
<p><img src="" alt="表1"><br><img src="" alt="表2"><br><img src="" alt="表3"><br><img src="" alt="表4"></p>

        
        
        <p class="article-more-link">
          
       </p>
    </div>
    <footer class="article-footer clearfix">
<div class="article-catetags">


  <div class="article-tags">
  
  <span></span> <a href="/tags/Python/">Python</a>
  </div>

</div>




<div class="comments-count">
	
</div>

</footer>


    </article>







</div>
      <div class="openaside"><a class="navbutton" href="#" title="Show Sidebar"></a></div>

<div id="asidepart">
<div class="closeaside"><a class="closebutton" href="#" title="Hide Sidebar"></a></div>
<aside class="clearfix">

  
<div class="github-card">
<p class="asidetitle">Github Card</p>
<div class="github-card" data-github="Golden-Pigeon" data-theme="medium"></div>
<script type="text/javascript" src="//cdn.jsdelivr.net/github-cards/latest/widget.js" ></script>
</div>



  

  
<div class="tagslist">
	<p class="asidetitle">Tags</p>
		<ul class="clearfix">
		
			
				<li><a href="/tags/Python/" title="Python">Python<sup>1</sup></a></li>
			
		
		</ul>
</div>


  <div class="linkslist">
  <p class="asidetitle">Links</p>
    <ul>
        
          <li>
            
            	<a href="https://github.com/Golden-Pigeon" target="_blank" title="我的github主页">我的github主页</a>
            
          </li>
        
    </ul>
</div>

  


  <div class="rsspart">
	<a href="/atom.xml" target="_blank" title="rss">RSS</a>
</div>

  <div class="weiboshow">
  <p class="asidetitle">Weibo</p>
    <iframe width="100%" height="119" class="share_self"  frameborder="0" scrolling="no" src="http://widget.weibo.com/weiboshow/index.php?language=&width=0&height=119&fansRow=2&ptype=1&speed=0&skin=9&isTitle=1&noborder=1&isWeibo=0&isFans=0&uid=&verifier=&dpc=1"></iframe>
</div>


</aside>
</div>
    </div>
    <footer><div id="footer" >
	
	<div class="line">
		<span></span>
		<div class="author"></div>
	</div>
	
	
	<section class="info">
		<p> Hello ,I&#39;m Golden Pigeon. <br/>
			Welcome to my blog!</p>
	</section>
	 
	<div class="social-font" class="clearfix">
		
		<a href="http://weibo.com/gold_pig2000" target="_blank" class="icon-weibo" title="微博"></a>
		
		
		<a href="https://github.com/Golden-Pigeon" target="_blank" class="icon-github" title="github"></a>
		
		
		
		
		
		
		
		<a href="http://www.zhihu.com/people/gold-pig-99" target="_blank" class="icon-zhihu" title="知乎"></a>
		
		
		
		<a href="mailto:GoldenPigeon2000@outlook.com" target="_blank" class="icon-email" title="Email Me"></a>
		
	</div>
			
		

		<p class="copyright">
		Powered by <a href="http://hexo.io" target="_blank" title="hexo">hexo</a> and Theme by <a href="https://github.com/wuchong/jacman" target="_blank" title="Jacman">Jacman</a> © 2020 
		
		<a href="/about" target="_blank" title="Golden Pigeon">Golden Pigeon</a>
		
		
		</p>
		<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js">
		</script>
		<span id="busuanzi_container_site_pv">
		本站总访问量<span id="busuanzi_value_site_pv"></span>次
		</span>
</div>
</footer>
    <script src="/js/jquery-2.0.3.min.js"></script>
<script src="/js/jquery.imagesloaded.min.js"></script>
<script src="/js/gallery.js"></script>
<script src="/js/jquery.qrcode-0.12.0.min.js"></script>

<script type="text/javascript">
$(document).ready(function(){ 
  $('.navbar').click(function(){
    $('header nav').toggleClass('shownav');
  });
  var myWidth = 0;
  function getSize(){
    if( typeof( window.innerWidth ) == 'number' ) {
      myWidth = window.innerWidth;
    } else if( document.documentElement && document.documentElement.clientWidth) {
      myWidth = document.documentElement.clientWidth;
    };
  };
  var m = $('#main'),
      a = $('#asidepart'),
      c = $('.closeaside'),
      o = $('.openaside');
  c.click(function(){
    a.addClass('fadeOut').css('display', 'none');
    o.css('display', 'block').addClass('fadeIn');
    m.addClass('moveMain');
  });
  o.click(function(){
    o.css('display', 'none').removeClass('beforeFadeIn');
    a.css('display', 'block').removeClass('fadeOut').addClass('fadeIn');      
    m.removeClass('moveMain');
  });
  $(window).scroll(function(){
    o.css("top",Math.max(80,260-$(this).scrollTop()));
  });
  
  $(window).resize(function(){
    getSize(); 
    if (myWidth >= 1024) {
      $('header nav').removeClass('shownav');
    }else{
      m.removeClass('moveMain');
      a.css('display', 'block').removeClass('fadeOut');
      o.css('display', 'none');
        
    }
  });
});
</script>












<link rel="stylesheet" href="/fancybox/jquery.fancybox.css" media="screen" type="text/css">
<script src="/fancybox/jquery.fancybox.pack.js"></script>
<script type="text/javascript">
$(document).ready(function(){ 
  $('.article-content').each(function(i){
    $(this).find('img').each(function(){
      if ($(this).parent().hasClass('fancybox')) return;
      var alt = this.alt;
      if (alt) $(this).after('<span class="caption">' + alt + '</span>');
      $(this).wrap('<a href="' + this.src + '" title="' + alt + '" class="fancybox"></a>');
    });
    $(this).find('.fancybox').each(function(){
      $(this).attr('rel', 'article' + i);
    });
  });
  if($.fancybox){
    $('.fancybox').fancybox();
  }
}); 
</script>



<!-- Analytics Begin -->





<!-- Analytics End -->

<!-- Totop Begin -->

	<div id="totop">
	<a title="Back to Top"><img src="/img/scrollup.png"/></a>
	</div>
	<script src="/js/totop.js"></script>

<!-- Totop End -->

<!-- MathJax Begin -->
<!-- mathjax config similar to math.stackexchange -->


<!-- MathJax End -->

<!-- Tiny_search Begin -->

<!-- Tiny_search End -->

  </body>
 </html>
